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Abstract. For the past few years, the authors have been working on the design and 
development of an online First Certificate in English (FCE) preparatory course and 
exam simulator in an attempt to provide a supplementary tool and resources for 
students aiming to achieve a B2 level of English. These materials have been designed 
in such a way that learners can follow and complete the activities either by being 
assessed by a remote tutor, or in a completely autonomous way. This makes the 
consideration, analysis and design of effective, efficient and constructive feedback 
an essential component in the whole design process. In this process, the concept of 
formative assessment needs to be taken into account from the initial stages of the 
design and development up to the last phase, and aims at validating the appropriateness 
of the contents and the resources which are part of the materials. This paper addresses 
feedback from a theoretical perspective while exploring its different roles from a 
practical point of view throughout these stages. The main objective is to present the 
pedagogical framework that has guided the whole design and development process, 
as well as to analyse the main parameters considered by the authors of these materials 
when designing and editing the feedback to be received by students after completing 
either automatically corrected or open input exercises and tasks. This entire process 
has been conductively aimed to offer learners the contents and all of the required 
additional tools, in particular corrective, personalised and formative feedback, in 
order to enrich the two main learning and assessment modalities: self-access and 
autonomous. Moreover, a description of the platform’s main utilities, which allow 
material writers to edit, monitor and assess branched feedback in accordance with the 
students’ performance, will be provided. 

Keywords: feedback, online English/FL materials, design, development, validation. 


* Contact author: anmarsae@gmail.com 


In L. Bradley & S. Thouesny (Eds.), CALL: Using, Learning, Knowing, EUROCALL Conference, Gothenburg, Sweden, 
22-25 August 2012, Proceedings (pp. 204-210). © Research-publishing.net Dublin 2012 


204 




Feedback: A Key Component in the Design, Development and Validation Stages of Online English... 


1. Introduction 

There is a need to explore students’ errors and mistakes so that material developers can 
anticipate them to avoid and reduce them or to offer an adequate solution when they 
have been already made. Ferris (2012) states that “corrective feedback is more effective 
when it is selective or focused” and that the emphasis should be on “identifying patterns 
of error that students could work on rather than marking every single error in sight” 
(p. 9), an idea to bear in mind when developing tools and resources to be used in 
autonomous or distance learning contexts. This is the case of the InGenio FCE Online 
Course & Tester , courseware for learners of English as a Foreign Language (EFL) 
developed to provide students at the Universitat Polilecnica de Valencia (UPV) with a 
supplementary tool to achieve a B2 level of English. In many cases, material developers 
explore the multiple ways in which language learning occurs and the features related to 
the causes that could explain why some students fail while others are able to attain the 
target level in the language being learnt. This fact is especially interesting as students 
are neither always aware of their progress nor are they able to describe the process that 
led them to that particular level of expertise (Ellis, 1997). 

2. Feedback 

Levy and Stockwell (2006) have presented research approaches in the field of Computer- 
Assisted Language Learning (CALL) based on the “patterns of learner behaviour as 
learners engage with online tasks and tutoring programs” (p. 153). A study on how 
students respond and interpret automatic feedback and on how they are able to correct 
their first response or performance can lead to important improvements. Pujola (2002), 
who dealt with the concept of “help facilities”, studied how students respond to those 
additional tools alongside the learning strategies which take part in the process too. 
This author distinguishes between the categories of assistance and guidance. Travis 
and Joseph (2009) consider that all the resources and tools relating to e-learning have 
“the potential to enhance the learning process, by offering more flexible access to the 
curriculum and providing opportunities for support outside the classroom” (p. 314). 
Arnold and Ducate (2011) reflect on the idea of dynamic evaluation and on how the line 
between instruction/formation and evaluation could be erased. 

Martinez Ruiz and Sauleda Pares (2001) present the concepts of formative evaluation 
versus summative evaluation. Formative evaluation would be aimed at understanding 
and improving the learning process by means of the effect of feedback when learners’ 
behaviour and answers have been predicted beforehand. Lara Ros (2002) studies 
the need to individualise, personalise, adjust, and adapt the entire process to comply 
with students’ particular requirements, an assumption related to the individualisation 
criteria aimed at responding to learning scenarios which are more and more focused on 
students’ specific needs and less on group anonymity and standardisation. 
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3. Computer assisted feedback and the feedback utility 

in the InGenio FCE Online Course <6 Tester 

An aspect which has been greatly agreed upon is the appropriateness of computer- 
aided evaluation, one of the most important reasons being the immediate reception of 
feedback by students (Lara Ros, 2002). Additionally, technology is often described as 
contributing very positively in every process, providing formative assessment as well 
as any other supplementary and supportive activity introduced to practice and improve 
specific contents. Lara Ros (2002) explores how the concept of evaluation, understood 
as help and orientation, is deeply related to the administration of constant feedback 
and reports, and refers to a study conducted by Stephens and Mascia (1996), for whom 
computer-assisted evaluation would enable students to know about their results at 
any point and detect problems, difficulties or drawbacks. These options would allow 
material developers and teachers alike to know which components are not working 
properly, as well as those general ideas or contents which are not being learned as 
predicted. 

According to Arneil and Holmes (2003), innovation and sophistication criteria can 
be introduced progressively if students’ production and performance are analysed in 
detail and compared with some possible correct answers. They consider that feedback 
could be immediate so that students could know whether their responses are correct 
immediately after completing the required tasks, enabling content and materials 
developers to introduce specific keys and hints as part of the activities and tasks 
implemented. Bangs (2003) concluded that feedback can be seen as one of the main 
components. Nevertheless, he alerts on the fact that an important part of the online 
materials, which are currently available, present some outstanding shortages, and sees 
the delivery of feedback as the factor which can lead to the conception of learning in 
an independent and individual way. Chapelle and Douglas (2006) also see feedback 
as an element to be taken into account when evaluating computer-assisted learning 
and assessment materials. Other authors such as Youngs, Ducate, and Arnold (2011) 
conceive interaction as a channel to provide students with very valuable feedback too. 
Hauck and Hurd (2005) present the important role played by tutorials, as many students 
need elements that could help them find similarities shared by CALL and face-to-face 
learning. 

The InGenio online authoring tool and learning management platform includes 
a feedback editing system in all the templates used to develop and edit exercises 
(Figure 1) (Gimeno San?. 2006). Feedback is approached in an intrinsic and extrinsic 
way and in an immediate or asynchronous way (Bangs, 2003). The aim is to provide a 
consistent pedagogical and didactic framework, enabling material developers to offer 
materials closely linked to students’ needs. Therefore, a student would not receive 
abrupt messages such as “wrong” or “wrong answer”, but a constructive response that 
would contribute dynamically to their linguistic development (Figure 2). 
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Figure 1. InGenio template to edit specific feedback 



Figure 2. Feedback per item as seen by the user 


3 http://camilleweb.upv.es - Unidad • Microsoft Internet Explorer 


Eurocall_pg4_MultipleChoice 1 


B0® 


IJ2SS / 


You should always be organised | 


you should mal e sure that the situat*on is under control 


This negative form use to male students confused 


Listen to the recording and decide which of the suggestions in the list below are 
mentioned by the person speaking. 

? DDo not try to guess what the other person will try to say. 

F Think about the purpose of the call: questions to ask and things to say. 

□ If someone calls when you are not ready, ask this person to call back later. 
E Prepare the desk: documents, paper, and computer files. 

? □ Check recent correspondence to know the situation. 

? El Do not take your diary: fix appointments in a separated piece of paper. 

| Evaluate 11 Show complete answer| 




■ ® ® 


Remember that you can refer to the audio book to read the script 





Eurocall_pg4_MulbpleChoicel 

Vi I r 

a 




•C Internet 


This is possible thanks to its branched feedback utility, which helps to provide feedback 
in accordance with the particular responses given. The provision of automatic feedback 
is possible across the different Course & Tester sections, with the only exception of the 
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tasks which require the student to produce written texts or record oral performances. 
The correction and rating of these tasks would require a human tutor to interact with 
the platform and rate the exercise by using the InGenio tutor interface and the rating 
options. When automatic correction is not possible, a model answer or answer key is 
provided so that students can compare their production with a sample of what would be 
expected from them (Figure 3) (Gimeno Sanz, 2002). 


Figure 3. Feedback in accordance with students’ efficiency rate and sample automatic 
messages 


3 http://caniilieweb.upv.es ln6enio Original content ... 



Perfect! If you have no doubts, jump to next section. 

This exerase is linked to page 3S and it is the first listening 
and comprehension exerase. 


3 http://camitleweb.upv.es ln6ento Original 


67% 

Good work and go ahead! 

This exerase is linked to page 3S and It Is the first listening 
and comprehension exerase. 



!1 http://camillcwcb.upv.es ln6cnlo Original content ... 


-3 Jt3l 


You should redo this exercise and practice more your 
listening. 

This exerase Is talked to page 35 and it is the first listening 
and comprehension exerase. 


frurocall pQ4 MultipleChoice 1 


Lull 




In September 2011 a group of 52 UPV students who had enrolled on a subject called 
Computer-Assisted English started working on the online B2 level courseware. The 
main component of this subject was the FCE Online Course & Tester, used as distance 
materials to be completed online, a learning modality to test the effectiveness of both 
the immediate and automatic feedback provided, as well as the asynchronous and 
personalised feedback offered to students. After the first semester, 27 students had 
completed the Course & Tester, and had responded to the final questionnaire designed 
to receive the students’ impressions, opinions and suggestions. This questionnaire, a 
key component in the validation stage, included several questions which can shed light 
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regarding the students’ perceptions on the feedback utilities available in the InGenio 
system or offered by the materials and by the tutors who had been monitoring the 
process. According to their responses to the initial questionnaire, a very high percentage 
of the students were worried about making mistakes (Results: Strongly agree: 21%, 
agree: 58%, neither agree nor disagree: 21%, disagree: 0%, strongly disagree: 0%). 
However, they felt less anxious after using the online materials. In accordance with their 
responses to the final questionnaire, rating through a 1 (disagreement) to 7 (agreement) 
Likert scale, almost all of them considered that the materials encouraged autonomous 
and independent learning (Results: 1: 0%, 2: 0%, 3: 7%, 4: 7%, 5: 37%, 6: 41%, 7: 
7%). Most of them reported that they enjoyed the fact of being able to self-assess their 
performance by accessing the progress reports (Results: 1: 0%, 2: 0%, 3: 4%, 4: 7%, 5: 
37%, 6: 30%, 7: 22%), and perceived the feedback they received as being useful and 
relevant (Results: 1:4%, 2: 0%, 3: 4%, 4: 15%, 5: 33%, 6: 26%, 7: 19%). 

4. Conclusions 

A representative number of studies have been conducted to observe and analyse 
students’ errors and mistakes in order to introduce corrective feedback so as to help 
students overcome them. Despite the negative connotations of the words “mistake” 
and “error”, their positive impact on language learning cannot be left aside. To study 
their origins or causes in depth could improve the learning process as well as offer 
feedback solutions to make it more effective and efficient. This idea has been taken 
into consideration during the design, development, implementation and validation 
stages of the FCE Online Course & Tester. The main goal has been to reinforce those 
resources as well as the learning and assessment modalities implemented. The idea of 
how important it is for students to receive materials suited to their own specific needs 
and demands has been addressed. Feedback has been approached from a theoretical 
perspective and a description of the utilities provided by InGenio, allowing material 
writers to edit, monitor and assess branched feedback in accordance with the students’ 
performance, has been presented. The students who have already used these materials 
have rated the effects of the feedback received very positively. 
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